System for obtaining clear endoscope images

ABSTRACT

An endoscopic system including an endoscope, a light source for the endoscope and at least one image sensor for capturing a plurality of images of a body cavity. The light source is configured to emit light from the endoscope and into the body cavity such that the light is reflected off of a plurality of locations in the body cavity. The system also includes a control system for controlling both the light source and the at least one image sensor to vary parameters of the light source and the at least one image sensor such that the plurality of images have underexposed and overexposed regions. The endoscopic system could alternatively include a variable-attenuator element device adjacent the image sensor and configured to be located between the body cavity and the image sensor for capturing a single clear image with the image sensor.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.14/159,003, filed Jan. 20, 2014, which claims the benefit of U.S.Provisional Application No. 61/779,693, filed Mar. 13, 2013, each ofwhich are incorporated herein by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates to a system for viewing clear images takenfrom an endoscope.

BACKGROUND OF THE INVENTION

An endoscope is a surgical tool designed to be placed inside a body inorder to provide a view of the interior portion of the body. Inendoscopic surgery, the endoscope is placed in the body at the locationat which it is necessary to perform a surgical procedure. Other surgicalinstruments are placed in the body at the surgical site. The surgeonviews the surgical site through the endoscope in order to manipulate theother surgical instruments to perform the desired surgical procedure.The development of endoscopes and their companion surgical instrumentshas made it possible to perform minimally invasive surgery thateliminates the need to make a large incision in the patient to gainaccess to the surgical site. Instead, during endoscopic surgery, smallopenings, called portals, are formed. One advantage of performingendoscopic surgery is that since the portions of the body that are cutare reduced, the portions of the body that need to heal after thesurgery are likewise reduced. Still another advantage of endoscopicsurgery is that it exposes less of the interior tissue of the patient'sbody to the open environment. This minimal opening of the patient's bodylessens the extent to which the patient's internal tissue and organs areopen to infection.

During endoscopic surgery, the environment of the body cavity may poseproblems relating to proper operation of the endoscope. For example, aproblem with images taken from within the body cavity is that portionsof the body cavity can be either underexposed or overexposed dependingon many factors including light intensities, exposure lengths andaperture sizes. The underexposed regions and overexposed regions of theimages can obscure important features which would be desirable todistinguish.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present invention are illustrated by wayof example and should not be construed as being limited to the specificembodiments depicted in the accompanying drawings, in which likereference numerals indicate similar elements.

FIG. 1 illustrates a perspective schematic view of an endoscopic systemaccording to the invention.

FIG. 2 is a schematic view of the endoscopic system according to thepresent invention for obtaining a clear image taken within a bodycavity.

FIG. 3 is a flow chart illustrating a method of obtaining the clearimage using the endoscopic system of the present invention.

FIG. 4 is a schematic view of a second endoscopic system according tothe present invention for obtaining the clear image taken within a bodycavity.

FIG. 5 is a flow chart illustrating a method of obtaining the clearimage using the second endoscopic system of the present invention.

FIG. 6 is a schematic view of a third endoscopic system according to thepresent invention for obtaining the clear image taken within a bodycavity.

FIG. 7 is a schematic view illustrating a control system of the thirdendoscopic system according to the present invention.

FIG. 8 is a schematic view of a fourth endoscopic system according tothe present invention for obtaining the clear image taken within a bodycavity.

FIG. 9 is a schematic view illustrating a control system of the fourthendoscopic system according to the present invention.

DETAILED DESCRIPTION

FIG. 1 illustrates an endoscopic system 20 including an endoscope 22, atransmission cable 46, and a light source console 28. The endoscope 22is defined by an elongated and generally hollow shaft 23 with a distalend 27 configured for insertion within a body cavity. The hollow shaft23 also has a proximal end 24 which mounts thereon an eyepiece 25 fittedto provide a viewing port through which the surgeon views the surgicalfield through a connection between the viewing port, a digital camera73, and a display screen 75 of a computer 100 (e.g., a person computeror other viewing system). A light port 58 may be connected with lightinputs to selectively transmit light to a target 50 (see FIG. 2) via theendoscope 22. In the illustrated embodiment, the light source console 28sends electromagnetic waves to the distal end 27 of the endoscope 22 toheat the same to prevent fogging.

The light source console 28 selectively provides electromagneticradiation as image capture light for use in the operating theater forilluminating the surgical field. In the present embodiment, thecandlepower of the image capture light emitted from the light sourceconsole 28 is selectively adjustable. Further, the light source console28 comprises a socket 43 to transmit the electromagnetic radiation fromthe light source console 28 to the endoscope 22 via intermediarydevices, such as the transmission cable 46.

The illustrated transmission cable 46 is configured to transmit lightfrom a proximal end 51 of the transmission cable 46 to a distal end 53of the transmission cable 46 attached to the light port 58. Thetransmission cable 46 can comprise an optical fiber or optical fiberssuited to transmit electromagnetic radiation via total internalreflection of such radiation within the fiber material. The proximal end51 and the distal end 53 include terminal geometries, such as plugs,conducive to receiving and emitting, respectively, electromagneticradiation.

As illustrated in FIG. 2, the body cavity 50 can have several areas thatare at different distances and angles from the distal end 27 of theendoscope 22. For example, the body cavity 50 can have a near area 56adjacent the distal end 27 of the endoscope 22, far area 54 at a bottomof the body cavity 50 and a middle area 52 between the near area 56 andthe far area 54. If the middle area 52 is properly light and focusedupon, the near area 56 can be overexposed (e.g., from the light beingtoo bright) and the far area 54 can be underexposed (e.g., from thelight being too dim). Therefore, an image captured by a digital capturesensor 102 within the camera 73 would not properly show all portions ofthe body cavity 50.

FIG. 3 illustrates a first method 101 for obtaining a clear image usingthe endoscopic system 20 of the present invention. In the first method101 for obtaining a clear image, multiple exposures 104 a, 104 b, 104 c,. . . 104 n are taken in sequential order at time t₀, t₁, t₂, . . .t_(n), respectively, with the different exposures 104 a, 104 b, 104 c, .. . 104 n being in varying degrees of underexposure, normal exposure andoverexposure. The normal exposure will obtain a clear view of the middlearea 52, the overexposure will obtain a clear view of the far area 54and the underexposure will obtain a clear view of the near area 56. Thevariances in exposure can be obtained by varying an intensity of a lightsource 103 (e.g., LED, xenon or other bulbs) within the light sourceconsole 28, by varying an exposure length (e.g., by electronic ormechanical shutter), varying an aperture size, varying a gain and/orother ways well known to those skilled in the art. In the illustratedexample, the computer 100 (or other control system) communicates withthe light source 103 to vary the intensity of the light source 103and/or communicates with the camera 73 to vary the exposure lengthand/or the aperture size. In the illustrated example, underexposure canbe accomplished by lowering the intensity of the light source 103, byshortening an exposure length and/or by reducing an aperture size.Likewise, overexposure can be accomplished by increasing the intensityof the light source 103, by lengthening an exposure length and/or byincreasing an aperture size.

In the illustrated example, once the series of multiple exposures 104 a,104 b, 104 c, . . . 104 n have been taken, the resulting images arealigned to compensate for differences in position of the endoscoperelative to the body cavity 50 during the difference in times t₀, t₁,t₂, . . . to in which the exposures 104 a, 104 b, 104 c, . . . 104 nwere taken. Properly aligning images is a process well known to thoseskilled in the art. Once the images have been aligned, a high dynamicrange (HDR) image is created at step 106. HDR image creation is wellknown to those skilled in the art. Tone mapping can then be applied tothe HDR image to produce a low dynamic range (LDR) image at step 108.HDR image and LDR image creation can be performed with the computer 100(or other control system) and the HDR image and/or the LDR image can beviewed on the display screen 75 (or any display device communicatingwith the computer 100 (or other control system).

FIG. 4 illustrates the endoscopic system 20 for use in a second method109 (see FIG. 5) for obtaining a clear image using the endoscopic system20 of the present invention. The endoscopic system 20 illustrated inFIG. 4 is identical to the endoscopic system 20 illustrated in FIG. 2,except that the camera 73′ includes a plurality of digital capturesensors 102 a, 102 b, 102 c . . . 102 n. FIG. 5 illustrates the secondmethod 109 for obtaining a clear image using the endoscopic system 20 ofthe present invention having the camera 73′ with the plurality ofdigital capture sensors 102 a, 102 b, 102 c . . . 102 n. In the firstmethod 109 for obtaining a clear image, multiple exposures 110 a, 110 b,110 c, . . . 110 n are taken all at the same time t₀, with each one ofthe digital capture sensors 102 a, 102 b, 102 c . . . 102 n taking oneof the exposures 110 a, 110 b, 110 c, . . . 110 n, respectively. All ofthe exposures 110 a, 110 b, 110 c, . . . 110 n will have the same lightintensity, such that only the normal exposures, underexposures andoverexposures are accomplished by varying an exposure length (e.g., byelectronic or mechanical shutter), varying an aperture size, varying again and/or other ways well known to those skilled in the art. In theillustrated example, the computer 100 (or other control system)communicates with the camera 73′ to vary the exposure length and/or theaperture size. In the illustrated example, underexposure can beaccomplished by shortening an exposure length and/or by reducing anaperture size. Likewise, overexposure can be accomplished by lengtheningan exposure length and/or by increasing an aperture size. It iscontemplated that the endoscopic system 20 could employ a beamsplitterfor sending the light to the plurality of digital capture sensors 102 a,102 b, 102 c . . . 102 n.

In the illustrated example, once the multiple exposures 110 a, 110 b,110 c, . . . 110 n have been taken, a high dynamic range (HDR) image iscreated at step 112 from the multiple exposures 110 a, 110 b, 110 c, . .. 110 n. HDR image creation is well known to those skilled in the art.Tone mapping can then be applied to the HDR image to produce a lowdynamic range (LDR) image at step 114. HDR image and LDR image creationcan be performed with the computer 100 (or other control system) and theHDR image and/or the LDR image can be viewed on the display screen 75(or any display device communicating with the computer 100 (or othercontrol system).

FIG. 6 illustrates a third endoscopic system 300 for obtaining a clearimage. The third endoscopic system 300 includes the endoscope 22 beingsupplied with light from a light source 302. The light source 302 couldbe located within the endoscope 22 or could be supplied to the endoscope22 via the transmission cable 46 and the light source console 28 asoutlined above. The light exits the distal end 27 of the endoscope 22 toimpinge upon the middle area 52, the far area 54 and the near area 56 ofthe body cavity 50. The third endoscopic system 300 also includes animage sensor 304 for capturing images. The image sensor 304 receivescontrol signals from a sensor control 306 and outputs sensor data 308including the captured images (see FIG. 7). The image sensor 304 couldbe a charge-coupled device (CCD) image sensor or a complementarymetal-oxide-semiconductor (CMOS) sensor. The endoscope 22 can alsoinclude a motion sensing device 310 (e.g., accelerometer or inertialmotion device) for sensing motion of the endoscope 22.

FIG. 7 illustrates a control system 311 of the third endoscopic system300 according to the present invention. The control system 311 of thethird endoscopic system 300 according to the present invention capturesa plurality of frames taken in sequential order (similar to the method101 as outlined in FIG. 3 above). The frames have different exposures tocreate a single output frame 313. The control system 311 controls thelight source 302 and the image sensor 304 to obtain the frames withdifferent exposures.

The illustrated light source 302 of the third endoscopic system 300 iscontrolled by a light control 312 of the control system 311 toselectively statically or dynamically alter the intensity (brightness)and/or pulse width modulation (duration) of the light source 302. Thelight control 312 receives an intensity (brightness) control signal 314and/or a pulse width modulation (duration) control signal 316 foraltering the intensity (brightness) and/or the pulse width modulation(duration) of the light source 302, respectively. The intensity(brightness) control signal 314 instructs the light source 302 toincrease or decrease intensity and the pulse width modulation (duration)control signal 316 instructs the light source 302 to alter the durationof the peak intensity of light at a desired intensity level. It iscontemplated that the light source 302 could include a mechanical orelectrical shuttering mechanism to control the pulse width modulation(duration) of the light source 302.

In the illustrated example, the image sensor 304 of the third endoscopicsystem 300 is controlled by a sensor control 306 of the control system311 to capture a desired frame or image. The sensor control 306 receivesa shutter (e.g., mechanical (e.g., rolling) or electric) control signal318, a gain (e.g., electric) control signal 320 and frame rate controlsignal 322 for controlling the image sensor 304 to obtain differentexposures of the frame or image. The shutter control signal 318instructs the sensor control 306 to control the shutter speed of theimage sensor 304. The gain control signal 320 instructs the sensorcontrol 306 to increase or decrease the gain of the image sensor 304.The frame rate control signal 322 instructs the sensor control 306 toincrease or decrease the frame rate of the image sensor 304.

The control system 311 of the third endoscopic system 300 creates aclear image by obtaining several frames from the image sensor 304. Thecontrol system 311 receives sensor (image or video) data 308 from theimage sensor 304 comprising multiple frames of different exposures intoa single output frame with high dynamic range. The exposures of themultiple frames can be different because the frames have any of thefollowing differences between the frames: different intensity of light,different duration of light, different shutter speed, different gainand/or different frame rate. In the illustrated example, the frames canbe buffered in a frame buffer 225 using alignment and motioncompensation data 227 to adjust for motion detected in the frame (e.g.,by using edge detection or re-alignment detection algorithms well knownto those skilled in the art) and/or to adjust for motion of the imagesensor 304 relative to the body cavity 50 sensed by the motion sensingdevice 310. The frames or images can also be transformed 330 usingintegrated light and exposure per pixel data 332 from the light control312 and the sensor control 306 (along with using the alignment andmotion compensation data 227 and the frame buffer 225) to maximize theamount of information provided to form the clear image using HDRtechniques without introducing distortion or unrealistic effects intothe output frame 313. It is also contemplated that the light control 312along with the transformed 330 frame and the alignment and motioncompensation data 227 could be improved using feedback 333.

In the illustrated example, the output frame 313 is therefore created bycontrolling both the light control 312 and the sensor control 306 alongwith using the sensor data 308, the alignment and motion compensationdata 227 and the integrated light and exposure per pixel data 332 toobtain the clear image. With the ability to control the light source302, the characteristics of the image sensor 304 including the timeduration imposed by the frame rate, the gain and/or the shutter speedare no longer limiting on the exposures of the frames.

FIG. 8 illustrates a fourth endoscopic system 400 for obtaining a clearimage. The fourth endoscopic system 400 includes the endoscope 22 beingsupplied with light from a light source 402. The light source 402 couldbe located within the endoscope 22 or could be supplied to the endoscope22 via the transmission cable 46 and the light source console 28 asoutlined above. The light exits the distal end 27 of the endoscope 22 toimpinge upon the middle area 52, the far area 54 and the near area 56 ofthe body cavity 50. The fourth endoscopic system 400 also includes animage sensor 404 for capturing images. The image sensor 404 receivescontrol signals from a sensor control 406 and outputs image data 408including the captured images (see FIG. 9). The image sensor 404 couldbe a charge-coupled device (CCD) image sensor or a complementarymetal-oxide-semiconductor (CMOS) sensor. The endoscope 22 also includesa variable-attenuator element device 410 located between the imagesensor 404 and the distal end 27 of the endoscope 22 (e.g., directly infront of the image sensor 404 or spaced from the image sensor 404). Ifthe variable-attenuator element device 410 is spaced from the imagesensor 404, it is contemplated that imaging optics could be used toalign elements of the variable-attenuator element device 410 with pixelsof the image sensor 404. Each element of the variable-attenuator elementdevice 410 allows a selected amount of light therethrough. It iscontemplated that the variable-attenuator element device 410 could be aliquid crystal display or any other device that includes a matrix ofvariable-attenuator elements that have a one to one mapping with pixelsof the image sensor 404 or one to a group or block of pixels of theimage sensor 404.

FIG. 9 illustrates a control system 411 of the fourth endoscopic system400 according to the present invention. The control system 411 of thefourth endoscopic system 400 according to the present invention capturesa single frame at a time, with each frame having a clear image. Theindividual frame will have different exposure areas to create a singleclear output frame 413. The control system 411 controls the light source402 and the image sensor 404 to obtain the frame.

The illustrated light source 402 of the fourth endoscopic system 400 iscontrolled by a light control 412 of the control system 411 toselectively statically or dynamically alter the intensity (brightness)and/or pulse width modulation (duration) of the light source 402. Thelight control 412 receives an intensity (brightness) control signal 414and/or a pulse width modulation (duration) control signal 416 foraltering the intensity (brightness) and/or the pulse width modulation(duration) of the light source 402, respectively. The intensity(brightness) control signal 414 instructs the light source 402 toincrease or decrease intensity and the pulse width modulation (duration)control signal 416 instructs the light source 402 to alter the durationof the peak intensity of light at a desired intensity level. It iscontemplated that the light source 402 could include a mechanical orelectrical shuttering mechanism to control the pulse width modulation(duration) of the light source 402.

In the illustrated example, the image sensor 404 of the fourthendoscopic system 400 is controlled by the sensor control 406 of thecontrol system 411 to capture a desired frame or image. The sensorcontrol 406 receives a shutter (e.g., mechanical (e.g., rolling) orelectric) control signal 418, a gain (e.g., electric) control signal 420and frame rate control signal 422 for controlling the image sensor 404to obtain a desired exposure for the frame or image. The shutter controlsignal 418 instructs the sensor control 406 to control the shutter speedof the image sensor 404. The gain control signal 420 instructs thesensor control 406 to increase or decrease the gain of the image sensor404. The frame rate control signal 422 instructs the sensor control 406to increase or decrease the frame rate of the image sensor 404.

The control system 411 of the fourth endoscopic system 400 creates aclear image by controlling the elements of the variable-attenuatorelement device 410 in order to maximize a brightness of the darkestportions of the body cavity 50 (e.g., those from the far area 54) whilealso eliminating or reducing the glare in the brighter portions of thebody cavity (e.g., those from the near area 56). The control system 411uses information in image data 408 from the image sensor 404 (e.g.,brightness of the pixel) along with the current attenuation at eachpixel of the image sensor 404 from the elements of thevariable-attenuator element device 410 (in the image analysis 430) tocontrol the light control 412 and the sensor control 406 via theintensity (brightness) control signal 414, the pulse width modulation(duration) control signal 416, the shutter control signal 418, the gaincontrol signal 420 and/or frame rate control signal 422 along withadjusting the elements of the variable-attenuator element device 410continuous on a frame by frame basis for image adjustment 432. The framecan be buffered in a frame buffer 425 communicating between the imageanalysis 430 and the image adjustment 432 using alignment and motioncompensation data to adjust for motion detected in the frame (e.g., byusing edge detection or re-alignment detection algorithms well known tothose skilled in the art) and/or to adjust for motion of the imagesensor 404 relative to the body cavity 50 sensed by a motion sensingdevice 409 (see FIG. 8).

In the illustrated example, the output frame 413 is therefore created bycontrolling the light control 412, the sensor control 406 along elementsof the variable-attenuator element device 410 such that each pixel ofthe image sensor 404 receives a desired amount of light in order to havea clear image. Therefore, multiple pixels in the image sensor 404 willreceive different levels of light. It is contemplated that the imagesensor 404 could have certain rows or columns of pixels dedicated tocalibrating the elements of the variable-attenuator element device 410.It is also contemplated that the light control 412, the sensor control406 and an element control 440 that controls the variable-attenuatorelement device 410 could be improved using feedback 450.

Although the present invention has been described with reference tospecific exemplary embodiments, it will be recognized that the inventionis not limited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. Accordingly, the specification and drawings are to be regardedin an illustrative sense rather than a restrictive sense. For example,the foregoing has involved surgical procedures specific to humans. Itwill be appreciated that the systems and methods described herein mayalso be applied to veterinary applications and non-biologicalapplications, for example for inspection of fluid chambers in industrialplants and transport devices. Moreover, it is contemplated that thelight source 103, 302, 402 can be located within the endoscope 22instead of within a light source console 28 separate from the lightsource 103, 302, 402 with the control system actively controlling thelight source 103, 302, 402 within the endoscope 22. Moreover, it iscontemplated that any of the first, second and third endoscopic devicescould additionally use the variable-attenuator element device 410.Although particular preferred embodiments of the invention have beendisclosed in detail for illustrative purposes, it will be recognizedthat variations or modifications of the disclosed apparatus, includingthe rearrangement of parts, lie within the scope of the presentinvention.

What is claimed is:
 1. A method of obtaining a clear endoscopic imagecomprising: providing an endoscope; providing a light source for theendoscope; providing an image sensor for capturing an image of a bodycavity, the image sensor having a plurality of image capture pixels;emitting light from the endoscope into the body cavity; reflecting thelight off of the body cavity; providing a variable-attenuator elementdevice between the body cavity and the image sensor, thevariable-attenuator element device having a plurality of lightattenuating elements; aligning the plurality of light attenuatingelements with the plurality of image capture pixels of the image sensor;controlling at least two of the plurality of light attenuating elementsof the variable-attenuator element device to attenuate some, but notall, of the light passing through the plurality of light attenuatingelements and into the plurality of image capture pixels; capturing theimage of the light with the plurality of image capture pixels of theimage sensor.
 2. An endoscopic system comprising: an endoscope; a lightsource for the endoscope; an image sensor for capturing an image of abody cavity, the image sensor having a plurality of image capturepixels; the light source being configured to emit light from theendoscope and into the body cavity such that the light is reflected offof the body cavity; a variable-attenuator element device adjacent theimage sensor and configured to be located between the body cavity andthe image sensor, the variable-attenuator element device having aplurality of light attenuating elements; a control system forcontrolling at least two of the plurality of light attenuating elementsof the variable-attenuator element device to attenuate some, but notall, of the light passing through the plurality of light attenuatingelements and into the plurality of image capture pixels.